# Observability [Logs, metrics, and indexing status]

## Logs

Ponder produces logs to help you understand and debug your application.

![Terminal logs gif](/logs-014.gif)

### Log level

There are two ways to configure the minimum log level. If specified, the environment variable takes precedence over the CLI flag.

- Set the `PONDER_LOG_LEVEL` environment variable
- Use the `--log-level <LEVEL>`, `-v` (debug) or `-vv` (trace) CLI option

```js [.env.local]
PONDER_LOG_LEVEL=trace
```

```bash [Terminal]
ponder dev --log-level warn
# or, use the shortcut flag for debug
ponder dev -v
```

#### Levels

| Log level        | Example                                                                    |
| :--------------- | :------------------------------------------------------------------------- |
| `silent`         |                                                                            |
| `error`          | Errors thrown in user code and other errors that will likely cause a crash |
| `warn`           | Malformed RPC data, reorgs, and other errors that will be retried          |
| `info` (default) | Indexing progress and key lifecycle events                                 |
| `debug`          | Internal updates                                                           |
| `trace`          | Database query logs, RPC request logs                                      |

#### User logs

Logs produced by your code (e.g. `console.log` statements in `ponder.config.ts` or indexing functions) will always be written to the console. Note that Ponder _does_ catch **errors** thrown by your code and emits an `error` log including the original error message and stack trace.

### Log format

Use the `--log-format <FORMAT>` CLI option to set the log format.

#### Pretty (default)

```bash [Terminal]
ponder start --log-format pretty
```

```bash [Output]
12:12:15.391 INFO  Indexed block chain=mainnet number=23569900 event_count=14 (23ms)
12:12:16.159 INFO  Indexed block chain=polygon number=77633630 event_count=0 (1ms)
12:12:16.174 INFO  Indexed block chain=optimism number=142386579 event_count=1 (4ms)
12:12:16.226 INFO  Indexed block chain=base number=36791294 event_count=9 (14ms)
12:12:18.068 INFO  Indexed block chain=optimism number=142386580 event_count=2 (8ms)
12:12:18.125 INFO  Indexed block chain=polygon number=77633631 event_count=0 (1ms)
12:12:18.188 INFO  Indexed block chain=base number=36791295 event_count=10 (16ms)
12:12:20.021 INFO  Indexed block chain=optimism number=142386581 event_count=0 (4ms)
```

#### JSON

```bash [Terminal]
ponder start --log-format json
```

The JSON log format emits newline-delimited JSON objects with required properties `level`, `time`, and `msg`. Most logs also include a `duration` property and other properties depending on the context.

```json [Output]
{"level":30,"time":1760372079306,"msg":"Indexed block","chain":"mainnet","chain_id":1,"number":23569912,"event_count":17,"duration":27.752416999996058}
{"level":30,"time":1760372080106,"msg":"Indexed block","chain":"polygon","chain_id":137,"number":77633702,"event_count":0,"duration":3.4684160000033444}
{"level":30,"time":1760372080122,"msg":"Indexed block","chain":"optimism","chain_id":10,"number":142386651,"event_count":0,"duration":2.3179999999993015}
{"level":30,"time":1760372080314,"msg":"Indexed block","chain":"base","chain_id":8453,"number":36791366,"event_count":10,"duration":18.320999999996275}
{"level":30,"time":1760372082131,"msg":"Indexed block","chain":"optimism","chain_id":10,"number":142386652,"event_count":0,"duration":3.074124999999185}
{"level":30,"time":1760372082258,"msg":"Indexed block","chain":"polygon","chain_id":137,"number":77633703,"event_count":0,"duration":1.7850829999952111}
{"level":30,"time":1760372082328,"msg":"Indexed block","chain":"base","chain_id":8453,"number":36791367,"event_count":4,"duration":9.394625000000815}
{"level":30,"time":1760372084153,"msg":"Indexed block","chain":"optimism","chain_id":10,"number":142386653,"event_count":0,"duration":2.679999999993015}
```

### Terminal UI

The dynamic terminal UI displays a useful summary of chain connection status, indexing function duration, and backfill progress.

```bash [Output]
Chains

│ Chain    │ Status │ Block     │ RPC (req/s) │
├──────────┼────────┼───────────┼─────────────┤
│ optimism │ live   │ 142388578 │         1.6 │
│ polygon  │ live   │  77635629 │        17.2 │
│ base     │ live   │  36793293 │         4.2 │
│ mainnet  │ live   │  23570232 │         7.5 │

Indexing (live)

│ Event         │ Count │ Duration (ms) │
├───────────────┼───────┼───────────────┤
│ WETH:Deposit  │   107 │         1.554 │

API endpoints
Live at http://localhost:42069
```

The terminal UI is disabled by default for `ponder start`. Use the `--disable-ui` CLI option to disable the UI for `ponder dev`.

```bash [Terminal]
ponder dev --disable-ui
```

## Metrics

Ponder apps publish Prometheus metrics at the `/metrics` path.

:::warning
Metrics are not part of the public API, so these are subject to change without notice. Do not rely on these metrics for anything important (yet).
:::

| name                                   | description                                                           | type      |
| :------------------------------------- | :-------------------------------------------------------------------- | --------- |
| ponder_indexing_total_seconds          | Total number of seconds required for indexing                         | gauge     |
| ponder_indexing_completed_seconds      | Number of seconds that have been completed                            | gauge     |
| ponder_indexing_completed_events       | Number of events that have been processed                             | gauge     |
| ponder_indexing_completed_timestamp    | Timestamp through which all events have been completed                | gauge     |
| ponder_indexing_function_duration      | Duration of indexing function execution                               | histogram |
| ponder_indexing_function_error_total   | Total number of errors encountered during indexing function execution | counter   |
| ponder_historical_start_timestamp      | Unix timestamp (ms) when the historical sync service started          | gauge     |
| ponder_historical_total_blocks         | Number of blocks required for the historical sync                     | gauge     |
| ponder_historical_cached_blocks        | Number of blocks that were found in the cache for the historical sync | gauge     |
| ponder_historical_completed_blocks     | Number of blocks that have been processed for the historical sync     | gauge     |
| ponder_realtime_is_connected           | Boolean (0 or 1) indicating if the realtime sync service is connected | gauge     |
| ponder_realtime_latest_block_number    | Block number of the latest synced block                               | gauge     |
| ponder_realtime_latest_block_timestamp | Block timestamp of the latest synced block                            | gauge     |
| ponder_realtime_reorg_total            | Count of how many re-orgs have occurred                               | counter   |
| ponder_database_method_duration        | Duration of database operations                                       | histogram |
| ponder_database_method_error_total     | Total number of errors encountered during database operations         | counter   |
| ponder_http_server_active_requests     | Number of active HTTP server requests                                 | gauge     |
| ponder_http_server_request_duration_ms | Duration of HTTP responses served by the server                       | histogram |
| ponder_http_server_request_size_bytes  | Size of HTTP requests received by the server                          | histogram |
| ponder_http_server_response_size_bytes | Size of HTTP responses served by the server                           | histogram |
| ponder_rpc_request_duration            | Duration of RPC requests                                              | histogram |
| ponder_rpc_request_error_total         | Total number of failed RPC requests                                   | counter   |
| ponder_postgres_pool_connections       | Gauge of current connections for PostgreSQL pools                     | gauge     |
| ponder_postgres_query_queue_size       | Current size of the query queue for PostgreSQL                        | gauge     |
| ponder_postgres_query_total            | Total number of queries processed by PostgreSQL                       | counter   |

## Indexing status

To check the indexing status of your app, use the `/status` endpoint or the `_meta` field in the GraphQL API.

### Usage

Use the indexing status to quickly confirm that Ponder is working as expected. You can also poll the status to confirm that a specific block number has been ingested by Ponder before refetching a query client-side (for example, in a form submit handler).

#### HTTP

```bash [Request]
curl http://localhost:42069/status
```

```json [Response]
{
  "mainnet": {
    "id": 1,
    "block": {
      "number": 20293450,
      "timestamp": 1720823759
    }
  },
  "base": {
    "id": 8453,
    "block": {
      "number": 17017206,
      "timestamp": 1720823759
    }
  }
}
```

#### GraphQL

```graphql [Query]
query {
  _meta {
    status
  }
}
```

```json [Result]
{
  "_meta": {
    "status": {
      "mainnet": {
        "id": 1,
        "block": {
          "number": 20293464,
          "timestamp": 1720823939
        }
      },
      "base": {
        "id": 8453,
        "block": null
      }
    }
  }
}
```

### API

The response object contains a property for each chain in your app with the following fields.

| field     |                       type                       | description                                                                 |
| :-------- | :----------------------------------------------: | :-------------------------------------------------------------------------- |
| **id**    |                     `number`                     | The chain ID.                                                               |
| **block** | `{ number: number; timestamp: number; } \| null` | The most recently indexed block, or `null` if the backfill is not complete. |
